#!/usr/bin/env python
# wh-words.py  -  Extracts 'wh' words from a provided corpus and outputs a sorted list of all wh word types.
# Tim Kane ()
#
# 12 March 2010


# Task 3.13
#
# Read in some text from a corpus, tokenize it, and print the list of all wh-word types that occur. (wh-words in English are used in 
# questions, relative clauses and exclamations: who, which, what, and so on.) Print them in order. Are any words duplicated in this list, 
# because of the presence of case distinctions or punctuation?
#

# Assumptions: I've assumed that *any* word beginning with 'wh' is of the form we require.
# I've not included any facility to determine if the sentence is a question / relative clause or exclamation. 
# And this in itself would not rule out secondary matches of 'wh' words within the same sentence.
#  eg: The sentence 'Why did he eat the whole thing?" would still match 'Why' and 'whole' even though 'whole' isn't the desired target word.



from __future__ import division
import nltk, re, pprint


def extract_wh(text):
    ''' This function takes textual input, and print a uniquely sorted list of all 'wh' words contained therin.'''

    #   The below initial test regex was borrowed from NLTK documentation.
    #   This is overkill for the task at hand.  A simpler regex is used below.

    #    pattern = r'''(?x)    # set flag to allow verbose regexps
    #        ([A-Z])+       
    #       | \w+(-\w+)*        # words with optional internal hyphens
    #       | [][.,;"'?():-_`]  # these are separate tokens
    #    '''


    # We assume that there are no 'wh' words that contain punctuation or non-alpha characters

    pattern = r'\w+'

    # Tokenize away

    tokens = nltk.regexp_tokenize(text, pattern)
    results = []

    # Determine the 'wh' words (this could have been handled in the above regex)

    for token in tokens:
        if re.search('^wh', token):
            results.append(token)

    # We now have a list of all 'wh' words, with many duplicates.
    # Duplicate removal can be be achieved by converting the list into a 'set'
    results = set(results)

    # Finally, we want to sort the set.  There is a 'sort' method for lists, so let's turn it back into a list again.

    results = list(results)
    results.sort()

    # And display what we've got

    for match in results:
        print match

    return


if __name__ == "__main__":
    
    corpus = (
""" I'd been waiting for the vampire for years when he walked into the bar.

Ever since vampires came out of the coffin (as they laughingly put it) four years ago, I'd hoped one would come to Bon Temps. We had all the other minorities in our little town -- why not the newest, the legally recognized undead? But rural northern Louisiana wasn't too tempting to vampires, apparently; on the other hand, New Orleans was a real center for them -- the whole Anne Rice thing, right?

It's not that long a drive from Bon Temps to New Orleans, and everyone who came into the bar said that if you threw a rock on a street corner you'd hit one. Though you better not.

But I was waiting for my own vampire.

You can tell I don't get out much. And it's not because I'm not pretty. I am. I'm blond and blue-eyed and twenty-five, and my legs are strong and my bosom is substantial, and I have a waspy waistline. I look good in the warm-weather waitress outfit Sam picked for us: black shorts, white T, white socks, black Nikes.

But I have a disability. That's how I try to think of it.

The bar patrons just say I'm crazy.

Either way, the result is that I almost never have a date. So little treats count a lot with me.

And he sat at one of my tables -- the vampire.

I knew immediately what he was. It amazed me when no one else turned around to stare. They couldn't tell! But to me, his skin had a little glow, and I just knew.

I could have danced with joy, and in fact I did do a little step right there by the bar. Sam Merlotte, my boss, looked up from the drink he was mixing and gave me a tiny smile. I grabbed my tray and pad and went over to the vampire's table. I hoped that my lipstick was still even and my ponytail was still neat. I'm kind of tense, and I could feel my smile yanking the corners of my mouth up.

He seemed lost in thought, and I had a chance to give him a good once-over before he looked up. He was a little under six feet, I estimated. He had thick brown hair, combed straight back and brushing his collar, and his long sideburns seemed curiously old-fashioned. He was pale, of course; hey, he was dead, if you believed the old tales. The politically correct theory, the one the vamps themselves publicly backed, had it that this guy was the victim of a virus that left him apparently dead for a couple of days and thereafter allergic to sunlight, silver, and garlic. The details depended on which newspaper you read. They were all full of vampire stuff these days.

Anyway, his lips were lovely, sharply sculpted, and he had arched dark brows. His nose swooped down right out of that arch, like a prince's in a Byzantine mosaic. When he finally looked up, I saw his eyes were even darker than his hair, and the whites were incredibly white.

"What can I get you?" I asked, happy almost beyond words.

He raised his eyebrows. "Do you have the bottled synthetic blood?" he asked.

"No, I'm so sorry! Sam's got some on order. Should be in next week."

"Then red wine, please," he said, and his voice was cool and clear, like a stream over smooth stones. I laughed out loud. It was too perfect.

"Don't mind, Sookie, mister, she's crazy," came a familiar voice from the booth against the wall. All my happiness deflated, though I could feel the smile still straining my lips. The vampire was staring at me, watching the life go out of my face.

"I'll get your wine right away," I said, and strode off, not even looking at Mack Rattray's smug face. He was there almost every night, he and his wife Denise. I called them the Rat Couple. They'd done their best to make me miserable since they'd moved into the rent trailer at Four Tracks Corner. I had hoped that they'd blow out of Bon Temps as suddenly as they'd blown in.

When they'd first come into Merlotte's, I'd very rudely listened in to their thoughts -- I know, pretty low-class of me. But I get bored like everyone else, and though I spend most of my time blocking out the thoughts of other people that try to pass through my brain, sometimes I just give in. So I knew some things about the Rattrays that maybe no one else did. For one thing, I knew they'd been in jail, though I didn't know why. For another, I'd read the nasty thoughts Mack Rattray had entertained about yours truly. And then I'd heard in Denise's thoughts that she'd abandoned a baby she'd had two years before, a baby that wasn't Mack's.

And they didn't tip, either.

Sam poured a glass of the house red wine, looking over at the vampire's table as he put it on my tray.

When Sam looked back at me, I could tell he too knew our new customer was undead. Sam's eyes are Paul Newman blue, as opposed to my own hazy blue gray. Sam is blond, too, but his hair is wiry and his blond is almost a sort of hot red gold. He is always a little sunburned, and though he looks slight in his clothes, I have seen him unload trucks with his shirt off, and he has plenty of upper body strength. I never listen to Sam's thoughts. He's my boss. I've had to quit jobs before because I found out things I didn't want to know about my boss.

But Sam didn't comment, he just gave me the wine. I checked the glass to make sure it was sparkly clean and made my way back to the vampire's table.

"Your wine, sir," I said ceremoniously and placed it carefully on the table exactly in front of him. He looked at me again, and I stared into his lovely eyes while I had the chance. "Enjoy," I said proudly. Behind me, Mack Rattray yelled, "Hey, Sookie! We need another pitcher of beer here!" I sighed and turned to take the empty pitcher from the Rats' table. Denise was in fine form tonight, I noticed, wearing a halter top and short shorts, her mess of brown hair floofing around her head in fashionable tangles. Denise wasn't truly pretty, but she was so flashy and confident that it took awhile to figure that out.

A little while later, to my dismay, I saw the Rattrays had moved over to the vampire's table. They were talking at him. I couldn't see that he was responding a lot, but he wasn't leaving either.

"Look at that!" I said disgustedly to Arlene, my fellow waitress. Arlene is redheaded and freckled and ten years older than me, and she's been married four times. She has two kids, and from time to time, I think she considers me her third.

"New guy, huh?" she said with small interest. Arlene is currently dating Rene Lenier, and though I can't see the attraction, she seems pretty satisfied. I think Rene was her second husband.

"Oh, he's a vampire," I said, just having to share my delight with someone.

"Really? Here? Well, just think," she said, smiling a little to show she appreciated my pleasure. "He can't be too bright, though, honey, if he's with the Rats. On the other hand, Denise is giving him quite a show."

I figured it out after Arlene made it plain to me; she's much better at sizing up sexual situations than I am due to her experience and my lack.

The vampire was hungry. I'd always heard that the synthetic blood the Japanese had developed kept vampires up to par as far as nutrition, but didn't really satisfy their hunger, which was why there were "Unfortunate Incidents" from time to time. (That was the vampire euphemism for the bloody slaying of a human.) And here was Denise Rattray, stroking her throat, turning her neck from side to side . . . what a bitch.

My brother, Jason, came into the bar, then, and sauntered over to give me a hug. He knows that women like a man who's good to his family and also kind to the disabled, so hugging me is a double whammy of recommendation. Not that Jason needs many more points than he has just by being himself. He's handsome. He can sure be mean, too, but most women seem quite willing to overlook that.

"Hey, sis, how's Gran?"

"She's okay, about the same. Come by to see."

"I will. Who's loose tonight?"

"Look for yourself." I noticed that when Jason began to glance around there was a flutter of female hands to hair, blouses, lips.

"Hey. I see DeeAnne. She free?"

"She's here with a trucker from Hammond. He's in the bathroom. Watch it."

Jason grinned at me, and I marvelled that other women could not see the selfishness of that smile. Even Arlene tucked in her T-shirt when Jason came in, and after four husbands she should have known a little about evaluating men. The other waitress I worked with, Dawn, tossed her hair and straightened her back to make her boobs stand out. Jason gave her an amiable wave. She pretended to sneer. She's on the outs with Jason, but she still wants him to notice her.

I got really busy -- everyone came to Merlotte's on Saturday night for some portion of the evening -- so I lost track of my vampire for a while. When I next had a moment to check on him, he was talking to Denise. Mack was looking at him with an expression so avid that I became worried.

I went closer to the table, staring at Mack. Finally, I let down my guard and listened.

Mack and Denise had been in jail for vampire draining.

Deeply upset, I nevertheless automatically carried a pitcher of beer and some glasses to a raucous table of four. Since vampire blood was supposed to temporarily relieve symptoms of illness and increase sexual potency, kind of like prednisone and Viagra rolled into one, there was a huge black market for genuine, undiluted vampire blood. Where there's a market there are suppliers; in this case, I'd just learned, the scummy Rat Couple. They'd formerly trapped vampires and drained them, selling the little vials of blood for as much as $200 apiece. It had been the drug of choice for at least two years now. Some buyers went crazy after drinking pure vampire blood, but that didn't slow the market any.

The drained vampire didn't last long, as a rule. The drainers left the vampires staked or simply dumped them out in the open. When the sun came up, that was all she wrote. From time to time, you read about the tables being turned when the vampire managed to get free. Then you got your dead drainers.

Now my vampire was getting up and leaving with the Rats. Mack met my eyes, and I saw him looking distinctly startled at the expression on my face. He turned away, shrugging me off like everyone else.

That made me mad. Really mad.

What should I do? While I struggled with myself, they were out the door. Would the vampire believe me if I ran after them, told him? No one else did. Or if by chance they did, they hated and feared me for reading the thoughts concealed in people's brains. Arlene had begged me to read her fourth husband's mind when he'd come in to pick her up one night because she was pretty certain he was thinking of leaving her and the kids, but I wouldn't because I wanted to keep the one friend I had. And even Arlene hadn't been able to ask me directly because that would be admitting I had this gift, this curse. People couldn't admit it. They had to think I was crazy. Which sometimes I almost was!

So I dithered, confused and frightened and angry, and then I knew I just had to act. I was goaded by the look Mack had given me -- as if I was negligible.

I slid down the bar to Jason, where he was sweeping DeeAnne off her feet. She didn't take much sweeping, popular opinion had it. The trucker from Hammond was glowering from her other side.

"Jason," I said urgently. He turned to give me a warning glare. "Listen, is that chain still in the back of the pickup?"

"Never leave home without it," he said lazily, his eyes scanning my face for signs of trouble. "You going to fight, Sookie?"

I smiled at him, so used to grinning that it was easy. "I sure hope not," I said cheerfully.

"Hey, you need help?" After all, he was my brother.

"No, thanks," I said, trying to sound reassuring. And I slipped over to Arlene. "Listen, I got to leave a little early. My tables are pretty thin, can you cover for me?" I didn't think I'd ever asked Arlene such a thing, though I'd covered for her many times. She, too, offered me help. "That's okay," I said. "I'll be back in if I can. If you clean my area, I'll do your trailer."

Arlene nodded her red mane enthusiastically.

I pointed to the employee door, to myself, and made my fingers walk, to tell Sam where I was going.

He nodded. He didn't look happy.

So out the back door I went, trying to make my feet quiet on the gravel. The employee parking lot is at the rear of the bar, through a door leading into the storeroom. The cook's car was there, and Arlene's, Dawn's, and mine. To my right, the east, Sam's pickup was sitting in front of his trailer.

I went out of the gravelled employee parking area onto the blacktop that surfaced the much larger customer lot to the west of the bar. Woods surrounded the clearing in which Merlotte's stood, and the edges of the parking lot were mostly gravel. Sam kept it well lit, and the surrealistic glare of the high, parking lot lights made everything look strange.

I saw the Rat Couple's dented red sports car, so I knew they were close.

I found Jason's truck at last. It was black with custom aqua and pink swirls on the sides. He sure did love to be noticed. I pulled myself up by the tailgate and rummaged around in the bed for his chain, a thick length of links that he carried in case of a fight. I looped it and carried it pressed to my body so it wouldn't chink.

I thought a second. The only halfway private spot to which the Rattrays could have lured the vampire was the end of the parking lot where the trees actually overhung the cars. So I crept in that direction, trying to move fast and low.

I paused every few seconds and listened. Soon I heard a groan and the faint sounds of voices. I snaked between the cars, and I spotted them right where I'd figured they'd be. The vampire was down on the ground on his back, his face contorted in agony, and the gleam of chains crisscrossed his wrists and ran down to his ankles. Silver. There were two little vials of blood already on the ground beside Denise's feet, and as I watched, she fixed a new Vacutainer to the needle. The tourniquet above his elbow dug cruelly into his arm.

Their backs were to me, and the vampire hadn't seen me yet. I loosened the coiled chain so a good three feet of it swung free. Who to attack first? They were both small and vicious.

I remembered Mack's contemptuous dismissal and the fact that he never left me a tip. Mack first.

I'd never actually been in a fight before. Somehow I was positively looking forward to it.

I leapt out from behind a pickup and swung the chain. It thwacked across Mack's back as he knelt beside his victim. He screamed and jumped up. After a glance, Denise set about getting the third Vacutainer plugged. Mack's hand dipped down to his boot and came up shining. I gulped. He had a knife in his hand.

"Uh-oh," I said, and grinned at him.

"You crazy bitch!" he screamed. He sounded like he was looking forward to using the knife. I was too involved to keep my mental guard up, and I had a clear flash of what Mack wanted to do to me. It drove me really crazy. I went for him with every intention of hurting him as badly as I could. But he was ready for me and jumped forward with the knife while I was swinging the chain. He sliced at my arm and just missed it. The chain, on its recoil, wrapped around his skinny neck like a lover. Mack's yell of triumph turned into a gurgle. He dropped the knife and clawed at the links with both hands. Losing air, he dropped to his knees on the rough pavement, yanking the chain from my hand.

Well, there went Jason's chain. I swooped down and scooped up Mack's knife, holding it like I knew how to use it. Denise had been lunging forward, looking like a redneck witch in the lines and shadows of the security lights.

She stopped in her tracks when she saw I had Mack's knife. She cursed and railed and said terrible things. I waited till she'd run down to say, "Get. Out. Now."

Denise stared holes of hate in my head. She tried to scoop up the vials of blood, but I hissed at her to leave them alone. So she pulled Mack to his feet. He was still making choking, gurgling sounds and holding the chain. Denise kind of dragged him along to their car and shoved him in through the passenger's side. Yanking some keys from her pocket, Denise threw herself in the driver's seat.

As I heard the engine roar into life, suddenly I realized that the Rats now had another weapon. Faster than I've ever moved, I ran to the vampire's head and panted, "Push with your feet!" I grabbed him under the arms and yanked back with all my might, and he caught on and braced his feet and shoved. We were just inside the tree line when the red car came roaring down at us. Denise missed us by less than a yard when she had to swerve to avoid hitting a pine. Then I heard the big motor of the Rats' car receding in the distance.

"Oh, wow," I breathed, and knelt by the vampire because my knees wouldn't hold me up any more. I breathed heavily for just a minute, trying to get hold of myself. The vampire moved a little, and I looked over. To my horror, I saw wisps of smoke coming up from his wrists where the silver touched them.

"Oh, you poor thing," I said, angry at myself for not caring for him instantly. Still trying to catch my breath, I began to unwind the thin bands of silver, which all seemed to be part of one very long chain. "Poor baby," I whispered, never thinking until later how incongruous that sounded. I have agile fingers, and I released his wrists pretty quickly. I wondered how the Rats had distracted him while they got into position to put them on, and I could feel myself reddening as I pictured it.

The vampire cradled his arms to his chest while I worked on the silver wrapped around his legs. His ankles had fared better since the drainers hadn't troubled to pull up his jeans legs and put the silver against his bare skin.

"I'm sorry I didn't get here faster," I said apologetically. "You'll feel better in a minute, right? Do you want me to leave?"

"No."

That made me feel pretty good until he added, "They might come back, and I can't fight yet." His cool voice was uneven, but I couldn't exactly say I'd heard him panting.

I made a sour face at him, and while he was recovering, I took a few precautions. I sat with my back to him, giving him some privacy. I know how unpleasant it is to be stared at when you're hurting. I hunkered down on the pavement, keeping watch on the parking lot. Several cars left, and others came in, but none came down to our end by the woods. By the movement of the air around me, I knew when the vampire had sat up.

He didn't speak right away. I turned my head to the left to look at him. He was closer than I'd thought. His big dark eyes looked into mine. His fangs had retracted; I was a little disappointed about that.

"Thank you," he said stiffly.

So he wasn't thrilled about being rescued by a woman. Typical guy.

Since he was being so ungracious, I felt I could do something rude, too, and I listened to him, opening my mind completely.

And I heard . . . nothing.

"Oh," I said, hearing the shock in my own voice, hardly knowing what I was saying. "I can't hear you."

"Thank you!" the vampire said, moving his lips exaggeratedly.

"No, no . . . I can hear you speak, but . . ." and in my excitement, I did something I ordinarily would never do, because it was pushy, and personal, and revealed I was disabled. I turned fully to him and put my hands on both sides of his white face, and I looked at him intently. I focused with all my energy. Nothing. It was like having to listen to the radio all the time, to stations you didn't get to select, and then suddenly tuning in to a wavelength you couldn't receive.

It was heaven.

His eyes were getting wider and darker, though he was holding absolutely still.

"Oh, excuse me," I said with a gasp of embarrassment. I snatched my hands away and resumed staring at the parking lot. I began babbling about Mack and Denise, all the time thinking how marvelous it would be to have a companion I could not hear unless he chose to speak out loud. How beautiful his silence was.

". . . so I figured I better come out here to see how you were," I concluded, and had no idea what I'd been saying.

"You came out here to rescue me. It was brave," he said in a voice so seductive it would have shivered DeeAnne right out of her red nylon panties.

"Now you cut that out," I said tartly, coming right down to earth with a thud.

He looked astonished for a whole second before his face returned to its white smoothness.

"Aren't you afraid to be alone with a hungry vampire?" he asked, something arch and yet dangerous running beneath the words.

"Nope."

"Are you assuming that since you came to my rescue that you're safe, that I harbor an ounce of sentimental feeling after all these years? Vampires often turn on those who trust them. We don't have human values, you know."

"A lot of humans turn on those who trust them," I pointed out. I can be practical. "I'm not a total fool." I held out my arm and turned my neck. While he'd been recovering, I'd been wrapping the Rats' chains around my neck and arms.

He shivered visibly.

"But there's a juicy artery in your groin," he said after a pause to regroup, his voice as slithery as a snake on a slide.

"Don't you talk dirty," I told him. "I won't listen to that."

Once again we looked at each other in silence. I was afraid I'd never see him again; after all, his first visit to Merlotte's hadn't exactly been a success. So I was trying to absorb every detail I could; I would treasure this encounter and rehash it for a long, long time. It was rare, a prize. I wanted to touch his skin again. I couldn't remember how it felt. But that would be going beyond some boundary of manners, and also maybe start him going on the seductive crap again.

"Would you like to drink the blood they collected?" he asked unexpectedly. "It would be a way for me to show my gratitude." He gestured at the stoppered vials lying on the blacktop. "My blood is supposed to improve your sex life and your health."

"I'm healthy as a horse," I told him honestly. "And I have no sex life to speak of. You do what you want with it."

"You could sell it," he suggested, but I thought he was just waiting to see what I'd say about that.

"I wouldn't touch it," I said, insulted.

"You're different," he said. "What are you?" He seemed to be going through a list of possibilities in his head from the way he was looking at me. To my pleasure, I could not hear a one of them.

"Well. I'm Sookie Stackhouse, and I'm a waitress," I told him. "What's your name?" I thought I could at least ask that without being presuming.

"Bill," he said.

Before I could stop myself, I rocked back onto my butt with laughter. "The vampire Bill!" I said. "I thought it might be Antoine, or Basil, or Langford! Bill!" I hadn't laughed so hard in a long time. "Well, see ya, Bill. I got to get back to work." I could feel the tense grin snap back into place when I thought of Merlotte's. I put my hand on Bill's shoulder and pushed up. It was rock hard, and I was on my feet so fast I had to stop myself from stumbling. I examined my socks to make sure their cuffs were exactly even, and I looked up and down my outfit to check for wear and tear during the fight with the Rats. I dusted off my bottom since I'd been sitting on the dirty pavement and gave Bill a wave as I started off across the parking lot.

It had been a stimulating evening, one with a lot of food for thought. I felt almost as cheerful as my smile when I considered it.

But Jason was going to be mighty angry about the chain. """ )


    extract_wh(corpus)

